Repository navigation
Run the FlatList maintainVisibleContentPosition E2E flows in Release only on iOS - #59638
Merged
meta-codesync[bot] merged 5 commits intoOct 30, 2026
Merged
Conversation
Each RNTester iOS flow had 5 attempts inside the job, and the whole job (both flavors, every flow, all setup) was retried up to 3 times on top, so a flow could run 15 times. The runner also stopped at the first flow that exhausted its attempts, hiding later failures. - Give each flow 2 attempts, rebooting the simulator before the retry. - Keep running the remaining flows after a failure and report every failed flow at the end, with a step summary. - Persist per-flow results between CI attempts (same format as the Android runner) so retries skip flows that already passed. - Only retry the workflow when a job stopped before running every flow. A flow that failed both attempts fails the run immediately. - Upload the recording, JUnit report and Maestro debug output for failed attempts only.
Test All had no concurrency group, so every push to a pull request left the previous run going to completion. Over Oct 27-29 about 40% of the macOS minutes spent on iOS E2E went to runs whose pull request had already received a newer push. Cancel in-progress runs of the same pull request. Push and workflow_dispatch runs get a unique group and are never cancelled.
The runner started a new `maestro test` process for every flow. Each one paid for JVM startup and an XCTest driver reinstall (~24s) plus stopping the screen recording (~3s): about 21 minutes of every ~75 minute RNTester job. - Run every pending flow in one `maestro test` invocation and read the per-flow results from its JUnit report. Flows missing from the report (Maestro crashed or timed out) count as failed. - Retry each failed flow on its own, on a rebooted simulator, with a screen recording. Passing runs don't produce recordings any more; failures keep Maestro's screenshots and debug output. - Add an `exclude-tags` input, and skip the `android-only` flows in RNTester. They only contain an Android-gated `runFlow`, but still paid the per-flow startup cost on iOS.
…only on iOS The 24 FlatList maintainVisibleContentPosition flows are half of the RNTester Maestro suite (~37 minutes per iOS job). Android already runs them against the release build only. On iOS about 88% of their failed attempts over Oct 27-29 were in Debug (133 vs 17). Rename their `android-release-only` tag to `release-only` and exclude it from the Debug job on both platforms.
This was referenced Oct 29, 2026
cipolleschi
changed the base branch from
cipolleschi/e2e-ios-3-single-invocation
to
main
October 30, 2026 11:43
|
@cipolleschi has imported this pull request. If you are a Meta employee, you can view this in D128581612. |
andrewdacenko
approved these changes
Oct 30, 2026
meta-codesync Bot
pushed a commit
that referenced
this pull request
Oct 30, 2026
**Stack** (iOS RNTester E2E cost, review bottom-up): 1. #59635 ← this PR 2. #59636 3. #59637 4. #59638 5. #59640 6. #59641 7. #59642 ## Summary: Part 1 of a stack that cuts the cost of the iOS RNTester E2E jobs (2 × macos-26-large, ~80 min each per run). Over Oct 27–29, 48% of Test All runs needed a workflow-level retry of these jobs and 44% of their macOS minutes went to those retries. The biggest offender was `sectionlist-viewability`: it failed ~88% of attempts for over a day, and the retries let some runs pass anyway. Each flow had 5 attempts inside the job, and the whole job (both flavors, every flow, all setup) was retried up to 3 times on top. That's up to 15 runs of a broken flow. The runner also stopped at the first flow that used up its attempts, so later failures stayed hidden. This PR: - Gives each flow 2 attempts, rebooting the simulator before the retry. The infrastructure failures in the logs (SpringBoard crash, "Unable to launch app", driver timeouts) left the simulator unusable for every later attempt on it. - Keeps running the remaining flows after a failure, reports every failed flow at the end and writes a step summary. - Persists per-flow results between CI attempts (same format as the Android runner), so retries skip flows that already passed. A flavor that already passed doesn't run again. - Only retries the workflow when a job stopped before running every flow (`retryable` output). A flow that failed both attempts fails the run immediately. - Uploads the recording, JUnit report and Maestro debug output for failed attempts only (previously only the last flow's recording per attempt number survived). The template app E2E shares the action and script. It gets the new in-job behavior (2 attempts, continue past failures) but keeps its own workflow retries unchanged. ### Results of the whole stack (CI, single runs) | | Debug job | Release job | | --- | ---: | ---: | | Before (this PR, flows unchanged, `macos-26-large`) | 89.8 min | 84.5 min | | After single Maestro invocation (#59637) | 53.0 min | 46.0 min | | Whole stack (#59642, `macos-26` arm64) | 22.0 min | 43.2 min | The Debug and Release jobs run in parallel. With the whole stack, the E2E stage takes ~43 min instead of ~90, on a runner that isn't billed for public repositories instead of two `macos-26-large` runners. Retries, cancellation of superseded runs (#59636) and the artifact gate (#59641) cut the cost of failing and outdated runs on top of that. ## Changelog: [INTERNAL] [CHANGED] - Retry failing iOS E2E flows once and stop rerunning real failures ## Test Plan: - `yarn jest .github/workflow-scripts/__tests__/maestro-ios-test.js`: 9 tests pass, covering continuing past failures, one retry with a simulator reboot, and skipping flows that passed in an earlier CI attempt. - Ran the script locally with Maestro 2.6.1 against `RNTesterApp-NewArch-Release` from run 40888922130, on an iPhone 17 Pro / iOS 26.5 simulator, with `appearance`, `button`, `image-wide-gamut` and a flow that always fails. The three passing flows ran once. The failing flow ran twice with a reboot in between, then the script exited 1. The state file recorded `passed`/`failed` and attempt counts, and `/tmp/MaestroLogs` only contained the failing flow's recording, report and debug output. - `actionlint` reports no new issues for the changed workflows. - CI on this PR exercises the workflow changes. Differential Revision: D128581149 Pull Request resolved: #59635
meta-codesync Bot
pushed a commit
that referenced
this pull request
Oct 30, 2026
**Stack** (iOS RNTester E2E cost, review bottom-up): 1. #59635 2. #59636 ← this PR 3. #59637 4. #59638 5. #59640 6. #59641 7. #59642 ## Summary: Part 2 of the iOS E2E cost stack. Test All has no `concurrency` group, so a new push to a pull request leaves the previous run going until it finishes. Phabricator-exported branches get pushed repeatedly (one had 8 runs in a day). Over Oct 27–29, ~40% of the macOS minutes spent on the iOS RNTester E2E jobs went to runs whose pull request had already received a newer push. This overlaps with the retry minutes addressed in part 1. Cancel in-progress runs of the same pull request. Push and `workflow_dispatch` runs each get a unique group (`github.run_id`), so runs on `main` and `*-stable` are never cancelled. `visual-compare-rntester.yml` (triggered by Test All's `workflow_run`) already ignores cancelled runs. ## Changelog: [INTERNAL] [CHANGED] - Cancel superseded Test All runs on pull requests ## Test Plan: - `actionlint` reports no new issues. - After this lands, pushing twice to a PR should cancel the first Test All run. Differential Revision: D128581367 Pull Request resolved: #59636
meta-codesync Bot
pushed a commit
that referenced
this pull request
Oct 30, 2026
**Stack** (iOS RNTester E2E cost, review bottom-up): 1. #59635 2. #59636 3. #59637 ← this PR 4. #59638 5. #59640 6. #59641 7. #59642 ## Summary: Part 3 of the iOS E2E cost stack. The runner started a new `maestro test` process for every flow. Each one paid for JVM startup and an XCTest driver reinstall (`--reinstall-driver` defaults to true): ~24s, plus ~3s to stop the per-flow screen recording. Across ~48 flows that's ~21 minutes of every ~75 minute job. - Run every pending flow in one `maestro test` invocation (`maestro test` takes multiple flow files) and read per-flow results from the JUnit report's `file` attribute. Flows missing from the report (Maestro crashed or timed out) count as failed. - Retry each failed flow on its own, on a rebooted simulator, with a screen recording. The first pass has no recording any more; failures keep Maestro's failure screenshots and debug output. - Add an `exclude-tags` input (same tag parsing as the Android runner) and skip the 4 `android-only` flows in RNTester. On iOS they only contain an Android-gated `runFlow`, but each still paid the per-flow startup cost. ## Changelog: [INTERNAL] [CHANGED] - Run iOS E2E flows in a single Maestro invocation ## Test Plan: - `yarn jest .github/workflow-scripts/__tests__/maestro-ios-test.js`: 11 tests pass, including parsing a Maestro JUnit report, tag filtering, and retrying only failed flows after the single run. - Ran locally with Maestro 2.6.1 against the CI-built Release app: `appearance` and `button` passed in a single Maestro run, `image-wide-gamut` was skipped as `android-only`, and only the always-failing flow was retried (on its own, after a reboot). The whole run took 2:23, versus 4:24 for the per-flow runner from part 1 on the same flows. - CI on this PR: Debug 53.0 min (44.8 min of flows), Release 46.0 min (40.2 min of flows), versus 89.8 / 84.5 min (82.3 / 76.2 min of flows) on #59635. All 44 flows passed in a single Maestro run in both flavors. Differential Revision: D128581505 Pull Request resolved: #59637
Collaborator
|
This pull request was successfully merged by @cipolleschi in 4665f25 When will my fix make it into a release? | How to file a pick request? |
meta-codesync Bot
pushed a commit
that referenced
this pull request
Oct 30, 2026
**Stack** (iOS RNTester E2E cost, review bottom-up): 1. #59635 2. #59636 3. #59637 4. #59638 5. #59640 ← this PR 6. #59641 7. #59642 ## Summary: Part 5 of the iOS E2E cost stack. The iOS E2E action ran all of its setup one step after another before touching the simulator, and `xcrun simctl install` then waited ~2 minutes for the simulator to finish booting. - Start booting the simulator as the first step (new `boot` subcommand of `maestro-ios.js`, which picks the simulator the same way as the test step). It then boots while Maestro, Java, the node modules and Metro are set up. - Drop the `brew install idb-companion` step (~16s). Maestro removed its IDB dependency on iOS in 1.x ("Removed IDB on iOS" in its changelog), and nothing in the 2.6.1 sources references it. - Drop the second Node.js setup inside the action. Both callers (RNTester and the template app) already set up Node.js in the same job. ## Changelog: [INTERNAL] [CHANGED] - Overlap simulator boot with the rest of the iOS E2E setup ## Test Plan: - `node .github/workflow-scripts/maestro-ios.js boot <model> <os>` locally: picks the configured simulator, boots it and returns. It accepts an already-booted simulator and auto-selects when given empty arguments, like the template app does. - `yarn jest .github/workflow-scripts/__tests__/maestro-ios-test.js` passes. - CI: the "Run tests" step should no longer spend ~2 minutes in `Installing app`. Differential Revision: D128581685 Pull Request resolved: #59640
meta-codesync Bot
pushed a commit
that referenced
this pull request
Oct 30, 2026
**Stack** (iOS RNTester E2E cost, review bottom-up): 1. #59635 2. #59636 3. #59637 4. #59638 5. #59640 6. #59641 ← this PR 7. #59642 ## Summary: Part 6 of the iOS E2E cost stack. `test_e2e_ios_rntester` only checks that `test_ios_rntester` succeeded, but `test_ios_rntester` uses `continue-on-error`, so it reports success even when the leg that uploads `RNTesterApp-NewArch-<flavor>` failed (the upload is its last step). In run 40709683649 the Release app was never uploaded and the E2E job still started, and retried, three times. Download the app as the first step after checkout, and treat a missing app as a non-retryable failure with an error that points to the build job, instead of an incomplete run that triggers the workflow retries. Gating in `test-all.yml` on the artifact itself would need `actions: read` to list artifacts, which the reusable workflow can't get under Test All's `contents: read`. ## Changelog: [INTERNAL] [CHANGED] - Fail iOS RNTester E2E straight away when the app artifact is missing ## Test Plan: - `actionlint` reports no new issues. - The missing-artifact path matches run 40709683649: `download-artifact` fails after its 3 attempts, the job is classified `failed`, and the run fails without workflow retries. Differential Revision: D128581762 Pull Request resolved: #59641
meta-codesync Bot
pushed a commit
that referenced
this pull request
Oct 30, 2026
**Stack** (iOS RNTester E2E cost, review bottom-up): 1. #59635 2. #59636 3. #59637 4. #59638 5. #59640 6. #59641 7. #59642 ← this PR ## Summary: Part 7 of the iOS E2E cost stack. The RNTester E2E jobs ran on `macos-26-large`, the Intel x86_64 runner, and the `-large` runners are billed at a premium. Run them on the standard arm64 `macos-26` runner instead. The simulator and app run natively on arm64, and standard GitHub-hosted runners aren't billed for public repositories. `RNTesterApp-NewArch-{Debug,Release}` are already universal (`lipo` shows x86_64 + arm64 for the app binary, `hermesvm`, `React` and `ReactNativeDependencies`), so the build job doesn't change. Measured on CI (table in the Test Plan), the 3-core standard runner is faster than `macos-26-large` despite having fewer cores: the Debug job took 22.0 min instead of 44.6, and the Release job 43.2 min instead of 55.0. The two jobs run in parallel, so the E2E stage now takes about as long as the Release job (~43 min instead of ~55). Billed minutes go from ~100 `-large` minutes per run to none. The only step that got slower is the Maestro download and install (~2 min more). If the standard runner ever turns out to be too slow, `macos-26-xlarge` (arm64, 5 cores) is the fallback. The template app E2E is unchanged. ## Changelog: [INTERNAL] [CHANGED] - Run iOS RNTester E2E tests on Apple Silicon runners ## Test Plan: - Checked the architectures of the CI artifacts from run 40888922130 with `lipo -info`. - Ran the RNTester Release artifact locally on an arm64 iPhone 17 Pro simulator with Maestro 2.6.1 (see parts 1 and 3). - CI, comparing the previous run of this PR (`macos-26`, arm64) with #59641 (the same stack, still on `macos-26-large`). Both runs still included the since-dropped deep-link change, so the comparison is fair. Single samples, so expect noise. | Job | `macos-26-large` (#59641) | `macos-26` (#59642) | | --- | ---: | ---: | | Debug (20 flows): total | 44.6 min | 22.0 min | | Debug: test step | 33.9 min | 16.1 min | | Release (44 flows): total | 55.0 min | 43.2 min | | Release: test step | 45.3 min | 37.2 min | | Maestro install | 1.2–1.4 min | 3.1–3.3 min | | yarn install | 6.5–6.7 min | 1.5–1.7 min | The flows run faster on arm64 even with 3 cores. The only consistently slower step is downloading and installing Maestro (~2 min more); yarn install timing varies between runs on both runners. - "GL pipe is running in software mode" still appears once per job on arm64 (vs 15–18 times per job on Intel), so I'm not claiming GPU rendering. Differential Revision: D128581888 Pull Request resolved: #59642
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stack (iOS RNTester E2E cost, review bottom-up):
Summary:
Part 4 of the iOS E2E cost stack.
The 24 FlatList
maintainVisibleContentPositionflows are half of the RNTester Maestro suite, ~37 minutes of each iOS job. Android already runs them against the release build only (#58140). They test scroll behavior that doesn't depend on the build flavor. Over Oct 27–29, ~88% of their failed attempts on iOS were in Debug (133 vs 17).Rename their
android-release-onlytag torelease-onlyand exclude it from the Debug job on both platforms. Android behavior is unchanged.Changelog:
[INTERNAL] [CHANGED] - Run the FlatList maintainVisibleContentPosition E2E flows in Release only on iOS
Test Plan:
yarn jest .github/workflow-scripts/__tests__/maestro-android-test.jspasses.actionlintreports no new issues.release-onlyflows; Release and Android are unchanged.